Critiquing the Tileworld: Agent Architectures, Planning Benchmarks, and Experimental Methodology

AI Planning for many years was concerned with solving problems in small, controlled domains like the Blocksworld, and testing the algorithms on small, pathological problems like the Sussman anomaly. Lately the eld has taken on two new goals: to apply planning techniques to more realistic worlds, and to nd a better way to validate the research eorts. One solution that is gaining popularity is to provide more-or-less realistic simulated worlds, in which one implements a planner and provides experimental results re BLOCKINecting its performance. The Tileworld system, reported in [Pollack and Ringuette 1990], confronts three interesting issues: what constitutes a good simulated testbed world for planning research, what constitutes a successful implementation of a planning architecture, and what constitutes experimental validation of such an implementation. This paper examines the program, claims, and experimental results contained in the Tileworld paper, and oers comments about the dicult issues facing a eld that is trying to apply its methods to more realistic problems and simultaneously trying to nd more rigorous and satisfying ways of measuring its progress.